Repository navigation
Conversation
tt/v41: TtV41Attention (q/kv stems without q_b_norm, V4.1 rope tables, sparse_sdpa over window + compressed KV with per-head sink and the ttMLA head-to-sequence reshard for 16 heads/chip, grouped O) and TtV41Block (lagged mHC, TtMoe with the V4.1 config). The compressed-KV path, window-KV FP8 QDQ and index-row assembly are explicit host fallbacks on the vendored reference modules. The test runs a 5-layer real-dims schedule standing for V4.1 layers 2,3,20,21,24 against the CPU oracle on LoudBox 2x4: deterministic; attention 0.988-0.994, MoE 0.996, block 0.997-0.998 (layer 0: 0.938, shown to be reference sensitivity to attention error at the first layer). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TtV41Attention takes an optional compute kernel config; the experiment runs the prototype attention chain for weight-dtype x fidelity variants. Result: bfp8/bf16 weights and HiFi2/HiFi4 differ by <=0.0005 PCC; the gap to the reference is its FP8 activation quantize-dequantize (dev-spec D-I). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TtV41HyperConnections applies a block's two mHC sites around caller-given sublayers with V4.1's lagged pre (attention uses the incoming pre, the FFN the one split before attention, the block returns the one split before its FFN), plus the post-model collapse with the last pre (no head mix). TtV41Block uses it as its only residual path. Against the reference at C=5120 on LoudBox 2x4, two chained blocks + final collapse reach >= 0.99999 PCC with synthetic and real layer 2/3 weights, bit-identical on repeat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One replicated natural-order row-major tensor per KV source: [128-row window slot | chunk window rows | compressed rows], so a layer's window KV and its source's compressed KV share one sparse_sdpa index space without copying the compressed rows. Chunks append only complete ratio groups of their valid length (padded tail writes nothing); per-SP-rank rows are gathered and written with slice_write. Per-layer 128-row window carries persist across chunks. Geometry goldens and a device test over two chunks and a padded tail pass on LoudBox 2x4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Real-weight blocks need the raw HF shards turned into the exact tensors the V4.1 device modules consume. weights.py is the host-only preparation phase: it reads tensors by name with one open per shard, dequantizes FP8 E4M3 32x32 blocks and MXFP4 routed experts (low nibble = even element, E8M0 per 32 along the input dim) to bf16 exactly, validates stored dtypes and shapes against the config, and returns the TtV41Block weights dict plus the role-specific compressor/indexer tensors and gate bias_vl. The checkpoint is pinned to HF revision dba1be0a by the git blob id of its index; $TT_V41_CHECKPOINT overrides the HF cache snapshot path. Device materialization and tensor caches stay with the modules that own the placement; the device test only proves real layer-2 attention weights place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ratio 1 is norm(wkv(x)) with the bf16 projection; ratio 2 pools each non-overlapping token pair with a per-channel softmax over fp32 wkv/wgate projections, then norm. A trailing incomplete pair yields no row and its fp32 projections are returned as the carry. Shared TP collectives move to tt/v41/ccl.py (attention uses them too). Against the reference Compressor at real dims on LoudBox 2x4: >= 0.99998 PCC for synthetic and real layer 2/20 weights, incl. odd length, carry and two chunks vs single shot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…chunk contract (§6) Block, cache and chunked device tests need trusted expected results for any subset of V4.1 layers, synthetic or checkpoint weights, without re-running the minutes-long CPU reference each time. oracle.py builds the vendored reference for a layer subset at real dims (roles, SWA-only ratio 0, Engram hash aligned to the full layout) or at small dims, captures per-block streams/sublayer I/O, the shared state each source publishes and the final state (window rings, ratio-2 carry, Engram tokens, DSpark rings) with forward hooks only, and caches results keyed by args, layer ids, seed, weights revision, tokenizer, prompt hash, reference-source digest, torch version and PACKAGE_VERSION. chunk_expectations slices the single-shot result into what a chunked (padded-tail) device run must match (graph.md §4); the reference itself is only ever run with start_pos 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reference quantize-dequantizes window KV and DSpark rings (FP8 e4m3, group 32, ue8m0), index keys and indexer queries (FP4 E2M1, group 32, ue8m0) and compressed KV (FP4 E2M1, group 16, e4m3 scale), and D-I proposes emulating its FP8 activation QDQ before the quantized GEMMs. Cache and score bars require the device to reproduce these values exactly given the same bf16 input, so this adds ttnn-composite ops (dev-spec D-H, F6.1a): exact group amax, power-of-two scales from the amax bit pattern (checked against fast_round_scale for every positive bf16), e4m3 RNE via exponent quantum + floor with exact tie resolution, and E2M1 rounding by exact threshold comparisons so the non-power-of-two e4m3 scale needs no exact division. Ops are row-local, so sharded or replicated mesh inputs work. Tests on the 2x4 mesh compare against kernel_cpu with torch.equal over random groups across ~40 decades, zero/floor groups, every e4m3/E2M1 half-way point and scale boundary (+-1 bf16 ulp), scale ties and saturation, W=512/128, S=2048, plus hand goldens and determinism. bf16 subnormal inputs (only relevant for FP4-ue8m0 groups with amax < ~2^-121) are documented as not reproduced: the device flushes subnormals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Queries are split over SP then TP, so each chip scores its contiguous queries with all 32 heads: no cross-chip score reduction, and the selection lands in the attention's head->sequence query layout. Scores use indexer_score_dsa with the V4.1 visibility rule applied explicitly (the kernel's causal offset is not per-chip under this split); candidate blocks and top-k use topk_large_indices with sentinel tails. qr gets the reference's FP8 activation QDQ before the exact wq_b; q and keys the FP4 QDQ. At real dims on LoudBox 2x4: index scores 0.9997, keys 0.9992, selection exact on tie-free scores for all three index roles, deterministic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scaling FP8/FP4 operands by their (power-of-two or exact) block scales before a single fp32 matmul is exact per element; only the fp32 summation order differs from the per-block loop, within the port's documented tolerance. The per-block Python loop made a 5-layer real-dims oracle exceed 90 minutes on a loaded host. CPU reference/oracle tests: 28/28 pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The vision tower (graph V1-V3) must turn one image's patches into the aligner rows the image span merge (10.2) consumes. TtV41Vision replicates the ~0.9 GB encoder on every device and runs it without collectives, so the replicated [T, 5120] output can be sliced by any SP/TP layout downstream. Semantics follow inference/vision.py exactly: rotate-half 2D RoPE via rotary_embedding_hf with the [h*f, w*f] angle vector duplicated over both halves; per-image bidirectional SDPA with cu_window_seqlens=[0, N, S] so tile/chunk padding never enters attention; the aligner's channel-major 3x3 unfold is replaced by a row-major reshape/pad/kh-slice gather plus a host permutation of w1's input columns (exact). Qwen-VL classes are not reused because they are bound to tt_transformers ModelArgs; their windowed SDPA and QKV head split are reused as mechanisms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, 384) The V4.1 gate matmul (640 tokens/chip, 5120/4 hidden, 384 experts) missed the tuned table and fell back to TTNN's default tiling, which measures 282 us per device (trace replay, BH) against 15.3 us for the swept 2D config (6x10 grid, per_core 2x2, in0_block_w 8, subblock 1x2); outputs are bit-identical. Only the V4.1 key is added, so no existing model's lookup changes. DeepSeek-V4.1 joins the gate PCC sweep, and the interleaved TP=1 case holds its deployed shape on the tuned entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndow rings (9.1, 9.2) Prefill's only DSpark state is each DSpark layer's window KV ring. TtV41DSpark reduces the tap-layer block input streams (mean over hc copies), runs main_proj (FP8 activation QDQ, row-parallel over TP, fp32 all-reduce) and main_norm on the tile-aligned rows around the last window positions, then per DSpark layer wkv -> kv_norm -> RoPE (theta 1e4, no YaRN, fp32) -> FP8 QDQ into ring slots pos % window. All steps are row-local, so seeding per chunk gives the single-shot ring (graph rule 6) with the ring as the only carry; padding rows write nothing (rule 8). Test vs the oracle (synthetic weights; real mtp shards are not downloaded): prompts shorter/longer than the window and a chunked prompt whose last window spans two chunks, small and real dims. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…acceptance (F1) TtV41Moe builds the in-tree TtMoe for V4.1: 384 experts, sqrtsoftplus DEVICE_FP32 gate, clamped SwiGLU, bfp8 or bfp4 routed experts (D-B). Two numerics fixes came out of real-weight acceptance: - The checkpoint's gate correction biases are about 10.76 +- 0.04, below the bf16 step at that magnitude (the device gate stores the bias in bf16), so top-6 recall fell to 0.90. The bias only selects experts, so the wrapper subtracts its mean; recall is back to 0.996. - D-I: the reference FP8-QDQs every expert GEMM input but not the gate's. TtMoe.forward takes an optional expert_x (no-op when omitted) so the gate routes on x while dispatch and the shared expert read fp8_qdq(x). Shared expert PCC 0.9986 -> 0.9994 (bar 0.999). test_moe_v41 runs the §6 oracle's layer-2 MoE input (5120 tokens) on 2x4 and 4x2, synthetic and checkpoint weights, bfp8 and bfp4, against the reference Gate/Expert parts (cached by moe_reference, torch only). All eight pass the G1 bars; test_dsv4_moe 4x2 rows reproduce their recorded PCCs with the TtMoe change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TtV41Embedding reuses TtParallelEmbedding (bf16 table, hidden sharded over TP), whose output already has the block layout, so no collective is needed; the lookup is exact. TtV41Head narrows the collapsed final hidden to the tile holding the requested row (M8: prefill needs one position), applies the distributed final RMSNorm (eps 1e-20) and TtLMHead column-parallel over the vocabulary with bf16 weights, fp32 activations and HiFi4/fp32 accumulation, returning fp32 logits like the reference ParallelHead. test_head_v41 (2x4, 4x2; synthetic and checkpoint embed/norm/head): embedding torch.equal to the reference; head logits PCC >= 0.99999 and identical top-1 on 10 rows spread over SP chips and tile offsets, including the oracle's own last-position logits; bit-identical repeats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…F6.4) TtV41Attention now runs the whole compressed-attention path on device over V41PrefillState: KV sources run compressor, index keys and the compressed KV write; index sources publish their top-k (the candidate source also its candidate blocks) and consumers reuse it; index rows are compacted with a top-k over validity and fed to sparse_sdpa on a head->sequence shard. FP8 activation QDQ precedes every quantized GEMM (dev-spec D-I). The state keeps the per-chunk selection and the ratio-2 carry; blocks take (state, length) and use TtV41Moe with an optional MoE weight cache. The host-fallback prototype and its tests are removed. Tests: tests/v41/test_block_v41.py runs layers 2,3,20,21,24 teacher-forced with device-produced state, single chunk and two chunks, at small dims (tests/v41/small_config.py, ~2 min per run) and real dims. Real weights, real dims: blocks 0.992-0.9987, caches >= 0.9987, deterministic. Small dims: layers 3-24 >= 0.9977 single and chunked; layer 2 (first, small residual, synthetic weights) 0.982 -- reference-reproduced sensitivity, open. Synthetic embeddings now use the checkpoint's RMS (0.12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lock ladder Synthetic norm weights and projection scales follow the released checkpoint (norm means by type, projection std as a multiple of fan_in^-0.5; measured on layers 0, 2, 20; user decision 2026-09-29): with unit norms the MoE output swamped a small residual and first blocks of a synthetic stack amplified tiny errors far beyond the real model. The oracle result key excludes testing.py for checkpoint-weight specs, so synthetic-init changes no longer invalidate real results. The small config uses 128 experts (32 hung the in-tree routing setup) and a real-like selected/visible ratio; the block test adds the SWA-only schedule. Small dims on LoudBox 2x4 (3m17s for 4 runs): all six block types in the schedules >= 0.9939, caches >= 0.9986, two chunks == one chunk, deterministic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shorter than a group Index rows are padded to a multiple of 32 so sparse_sdpa always gets a valid k chunk (a 3-token prompt produced 144 rows). The block test covers a 500-token prompt as 256 + 244 (padded tail, graph.md rule 8) and a 3-token prompt (empty or single compressed row, empty top-k) for the sharing stack and the SWA-only layer. Small dims: all pass, blocks >= 0.9943. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e (F5) Engram (layers 1, 14) adds an n-gram table lookup into the 4 residual streams before the block. Per dev-spec D-D the hash and the table lookup run on the host (token-only, prefetchable): V41EngramHash reproduces the reference NgramHashState with the last 3 compressed ids carried across chunks (graph.md rule 5; padded tail never hashed, rule 8) and V41EngramTable dequantizes the FP8/E8M0 rows as ParallelEngramEmbedding does. TtV41Engram runs wkv (FP8 activation QDQ, columns permuted so each TP chip holds its hidden slice of the 4 keys and the value) and the per-copy normalized-dot gate on the packed fp32 streams; the per-copy sums are TP partials, all-gathered and added in chip order so every chip computes the same gate. HiFi4 + fp32 accumulation on wkv: HiFi2 gives update PCC 0.99985 vs 0.99999 (bfp8 vs bf16 weights barely matter). Tests vs the reference (cached CPU results in engram_reference.py): hash ids and lookup bit-exact incl. chunked history and an image span across a chunk boundary; device update >= 0.999 per copy, image rows unchanged, repeat bit-identical, at small dims and at the production chunk 5120. Real Engram weights/tables (shards 47/48) are not downloaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TtV41Transformer runs a request chunk by chunk: embedding, 4-stream expansion, one-hot pre-mix, the layers in order over V41PrefillState (DSpark taps and per-chunk ring seeding when configured), and after the last chunk the final collapse, norm and LM head on the last real token. It owns the MoE device weight cache (per-layer completion marker; host MoE weights are skipped when cached). Engram layers are rejected until F5 is wired. The block test keys its weight cache by weight identity. Small dims on LoudBox 2x4 (layers 0,2,3,20,21,24; 59 s for 3 cases): last-token logits PCC 0.986 / 0.993 / 0.996 for one chunk, two chunks and a padded prompt, top-1 match, bit-identical repeats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Engram layers hash each chunk's valid tokens (carrying the n-gram history across chunks), prepare their rows, and update the streams before their block. The transformer test adds an Engram schedule (layers 0,1,2,3) and checks top-1 only when the reference's own margin exceeds the device's logit error (synthetic heads give near ties; G1 gates stacks on PCC). Small dims, 6 cases in 1.5 min: logits PCC 0.986-0.998, deterministic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Synthetic production-shape index-source blocks are gated at the real-weight bar (0.98; user decision 2026-09-29): FP4-quantized top-k selection disagrees intrinsically between bf16-accurate implementations (reference self-recall 0.993 at bf16-level input perturbation) and synthetic attention is ~7x more sensitive to those flips than real. Production shape on LoudBox 2x4, layers 2,3,20,21,24, one and two chunks: synthetic 0.9858-0.9993, real 0.9928-0.9995, caches >= 0.9988, deterministic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
F6.1b asks for end-to-end cache contents >= 0.998 for every cache the prefill writes. The block test checked compressed KV and index-K but not the window KV; it now compares each layer's window carry (the last 128 positions' post-RoPE FP8-QDQ rows, right-aligned) with the oracle's window KV. Small dims 12/12 (>= 0.9992), production synthetic + real (>= 0.9989). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remaining F5 acceptance: the table lookup on device and the host-vs-device decision. Both table placements now dequantize on device from the stored bytes, chosen at construction by the table object passed to TtV41Engram: - host table (V41EngramTable): rows packed once (pack_rows: e4m3 bytes and E8M0 scales rearranged into uint16 containers, 320 B/row); per chunk one host row gather, each chip gets 1/8 of the lookups (1 B/value instead of TP-replicated bf16), decodes them and all-gathers over TP. - device table (TtV41EngramTable): packed rows row-sharded over the 8 chips plus a zero row per shard; only per-chip local row ids go up; every chip gathers (ttnn.embedding on the bf16-carried containers), the byte values (0..255, exact in bf16) are reduce-scattered over SP then TP, decoded and all-gathered over TP. Summing 16-bit containers in the reduction lost the low byte, hence bytes. The device decode is exact vs the reference dequant as values (a table of every finite e4m3 code, scales 2^-30..2^30); the device writes e4m3 -0 as +0. Measured at a 5120-token chunk (synced wall): lookup host 7 ms + device 12 ms with the device table, host 35 ms + device 7 ms with the packed host table, host 73-98 ms with the previous bf16 upload. The device table needs 15.4 GB/chip per table on 2x4, so LoudBox keeps the host table; see the bead for the decision record. oracle.synthetic_engram_rows draws rows in fp32 regardless of the caller's default dtype (under set_dtype(bf16) it produced different rows); fp32-default results are unchanged, so cached oracles stay valid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DeepSeek-V4.1 attends over one 512-dim KV row with RoPE inside it; its stage-2 cache format (epic KV FORMAT) is scaled FP8 over all 512 dims: [512 FP8 | 4 FP32 scales], 528 B per row. The shared ops assumed the DSA 512+64 geometry: - pack_scaled_fp8_kv_cache: rope input is optional; latent / scale / RoPE widths come from the inputs (latent a multiple of 128) using the sparse_sdpa_common.hpp row geometry. The program hash keys on RoPE presence and shape; the RoPE copy compiles out when absent (placeholder accessor args). - sparse_sdpa SCALED_FP8 with K_DIM == v_dim: no RoPE CBs, the RoPE tilize and the RoPE partial QK matmul are compiled out; the RoPE-offset alignment check applies only when a RoPE tail exists. Changes sit inside the scaled_kv branch; BF16 and FP8_E4M3 paths are unchanged. - MlaKvCacheGeometry accepts rope_dim 0. Tests: new rope-less pack cases (512 and one-scale 128 rows) and sparse_sdpa scaled-FP8-without-RoPE cases (512 at H 32/64, 128). Consumers re-run: post-commit test_sparse_sdpa.py (50/50), nightly pack test (6/6), sparse KV cache contract (15/15), sparse_mla SCALED_FP8 chunked/pad-overflow/accuracy on 2x4 (5/5). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…me format (F6.2)
Stage 2 of the epic KV FORMAT: the compressed layers' KV tensors (window slot, chunk
window rows and compressed rows share one sparse_sdpa call, so one format) are stored
as [512 FP8 e4m3 | 4 FP32 power-of-two scales] per row, quantized per 128 dims (RoPE
included) from the unrounded post-RoPE KV, never from the FP4/FP8-QDQ values.
- V41PrefillState(kv_format=MlaKvCacheFormat.BF16_RM | SCALED_FP8), default BF16_RM
(unchanged behavior). The state encodes rows it writes (per_token_cast_to_fp8 +
rope-less pack) and decodes them for host checks (to_host). Ratio-0 (SWA-only)
layers stay BF16 in both stages: they hold the reference FP8 block-32 window values
exactly and share no tensor with compressed rows. Index-K stays BF16.
- TtV41Attention applies the reference QDQ only at the BF16 stage and passes the
layer's format to sparse_sdpa.
- test_block_v41 gains kv_format {bf16, scaled_fp8}. At scaled_fp8 compressed KV is
compared with the reference's compressed KV before its FP4 QDQ (derived from the
reference compressor on the recorded attention input; the test asserts its FP4 QDQ
reproduces the oracle rows bit for bit). The PCC against the FP4-QDQ oracle rows is
reported only: it is ~0.995 by construction (FP4 vs unrounded), below 0.998.
Also includes the coordinator's DSpark ring slot (dspark_rings) in cache.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TtV41Moe sized the dispatch buffer with capacity factor 2. Repetitive real text concentrates routing on one chip's experts; the overflowing tokens were dropped silently and in arrival order, so a small Engram stack on a real-text prompt was neither correct nor bit-identical across repeated prefills. Top-k (6) is the only factor that bounds the worst case (a token can land all its experts on one chip); in-tree DeepSeek-V3 prefill uses its top-k (8) the same way. With it the repeats are bit-identical again (diagnosis: evidence/F9-drift-diagnosis/diag_cap.log). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eference noise floor (F9) Free-running V4.1 stacks are chaotic: top-k selection and MoE routing flip under tiny input error (real layer 20's MoE keeps the same experts on 77% of rows at input PCC 0.9987; the reference's own last-token logits move to PCC 0.968 under 0.1% embedding noise), so an absolute bar on one token's logits measured the flips, not the device. User decisions 2026-09-29: - prompts are real text (oracle.text_tokens: A Tale of Two Cities, the tt_transformers accuracy book); a prefill is teacher-forced by construction, so the last 256 positions are scored (repo convention, e.g. DeepSeek-V3 teacher-forced top-1/top-5); - bar = the reference's worst self-agreement over 5 seeds (oracle.tail_logits with noise) under 4.5% noise on every attention and MoE output (the error a component bar of 0.999 allows) plus 0.4% on every attention input (where the reference's layer-20 attention matches the device's), minus sampling margins; top-1 agreement, top-5 recall and logits PCC; - DSpark rings from free-running streams are reported, not gated (the 0.998 ring bar applies with teacher-forced taps, gated in test_dspark). Device: TtV41Transformer.prefill(tokens, logit_positions) returns the logits of the last N prompt positions (TtV41Head.rows_to_host, one projection per 32-row tile) and keeps the DSpark rings in V41PrefillState. Tests: production-shape stack (layers 0 2 3 20 21 24, S=2048, synthetic + real), small DSpark schedule (20, 36-39), stage timing logs, identity-keyed device weight caches at small dims. Small 9/9 + oracle CPU test pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orted meshes rejected (11.1) Every V4.1 module already derives SP/TP from mesh_device.shape, but nothing stated which meshes the layer can run on: an unsupported mesh failed deep in an op (or, for MoE, silently floored experts // chips). tt/v41/layout.py (V41MeshLayout) now owns the SP=0 / TP=1 axis convention and the layout requirements: - model (checked by TtV41Block): hidden/tp whole tiles, heads % tp, heads % 32 (sparse_sdpa after the head->sequence all-to-all), O_GROUPS % tp, routed experts % chips; - chunk (checked by V41PrefillState): chunk % (32*sp*tp), whole-tile query rows per chip for the head->sequence attention and the indexer query split (the SP rule 2*32*sp stays with V41CacheGeometry). Both accept LoudBox 2x4 / 4x2 and Galaxy 8x4 / 4x8 at real and small dims; 2x4 behavior is unchanged (all existing chunks satisfy the new rule). The perf model no longer defaults to the LoudBox 2x4 layout (every caller passes one). Tests: tests/v41/test_v41_layout.py (CPU, parametrized over the four layouts plus rejection cases); 2x4 block + transformer small cases and perf-model tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he prototype oracle (§6) The vision test built its own expected values on every run (up to ~2 min of CPU per 1920x1080 case). The oracle package now owns them: vision_oracle runs the reference encoder (vision.ViT -> vision.Aligner, as Transformer.encode_image) on synthetic (per-tensor seeded) or checkpoint (shard 1, as stored) weights and caches the result keyed by args, weights source, patch digest, grid, vision.py digest, torch version and PACKAGE_VERSION. Test inputs (synthetic PNGs through image_processor, direct patch grids) come from the package too. Existing oracle results and cache keys are unchanged. The prototype oracle's schedule is real_spec((2, 3, 20, 21, 24), 2048, candidate_topk_blocks=96); test_indexer now uses it (V4.1 layer ids). device_weights / dequant are a device-layout helper, not an oracle: moved to tests/v41/reference_weights.py, with include_moe=False to skip the routed-expert FP4 dequantization for callers whose device MoE tensors are cached. prototype_oracle.py keeps only a device_weights re-export until test_block_v41 / test_transformer_v41 switch imports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Several whole prompts concatenated in one generic chunk (contiguous from row 0, ratio-aligned lengths, the chunk at start 0), each isolated from the others by the generic chunk inputs it already takes plus three host-side extensions: V41ChunkTables fills RoPE rows per segment (local positions), keeps a resident window-row table per layout (-1 where a query would read the previous prompt), and adds a per-query lower bound to the indexer masks (the first compressed row of the query's segment, one lt and one logical_or per mask) so no selected row lies before the segment; the attention compacts every row of the chip (a segment can start anywhere on it); the state writes each segment's window carry into its own buffer (segment_carry) beside the chunk's; each prompt is Engram- hashed from an empty history; the head scores each segment's last row. TtV41Transformer.prefill_packed runs one such chunk eagerly and returns one logits row per prompt. With the flag off (the default) no graph changes: the extra mask ops and the layout table exist only while a packed layout is being run. tests/v41/test_v41_pack.py compares each segment, bit for bit, with the unpacked prefill of the same prompt on the same geometry (logits, cache rows in segment coordinates, carries, selections), mutates the earlier neighbour, runs two negative controls (a cross-segment window table, the lower bound dropped), checks the flag-off state, and times the single-prompt chunk and each pack (eager, and captured and replayed with V41_PACK_TRACE=1). Action-Id: ds41/optimize/233
The packed-chunk test reads every chip copy as [chips, 1, rows, W] (a stacked 5-D readback sliced the wrong axis for the row-major caches), records per key the differing rows, compares each selection as a set (sorted ids) beside its score order, probes each block's norm, attention and FFN outputs and, one level down, the attention's projections, TP all-reduces and RoPE (V41_PACK_PROBE=1|2), and packs prompt B behind a first segment of V41_PACK_OFFSETS lengths. Its assertions are the isolation contract (the selection ranges, the flag off, segment 0, every segment under its neighbour's mutation, determinism, the negative controls); a later segment against its single run at row 0 is reported and asserted only under V41_PACK_STRICT=1, since it differs by the TP all-reduce's row-dependent summation order and the top-k's tie order. V41ChunkTables records the flag (_pack) so a state without a ratio > 0 layer can run a pack too. Action-Id: ds41/optimize/233
Per-segment joins behind TT_V41_PACK with TT_V41_PACK_LAYOUTS: V41DecodePool.join_inputs / join_ops / join take segment=(offset, length, g) (the compressed rows and index keys copied from offset / ratio, the index covering length / ratio rows, the segment's own window carries, no compressor carry), TtV41Decoder.join / capture_join carry it (the user position is the segment's length: the masks' and the c1 live-user index's bound; the capture key adds the offset and carry index), V41PrefillState allocates pack_segments carries at construction and the layouts' tables on request, and the generator builds one packed state at the first bucket, runs the FIFO prefix of a call's whole prompts that forms a listed layout as one packed chunk (eager; its rows pending on their segments, joined at their first decode step or when the next pack needs the state), prebuilds the layouts' totals' padding configs and runs one eager pack with its segment joins in the compile pass. tests/v41/test_v41_pack_join.py is the decode-token gate. Action-Id: ds41/optimize/233
V41PackTrace: the packed chunk of one layout captured once on the packed state after the eager warm-up runs and replayed for every pack of that layout (the token ids, Engram inputs and the layout's fills copied before the replay; the k scored rows read after it; the replay's device time in last["wait"]). The generator's trace pass captures every listed layout and then the joins of its segments into every slot at the segments' width, after the prefill states' joins and generic captures and before the decode warm-up's trace pass; TT_V41_PACK_TRACE=0 keeps the packs eager. Every persistent tensor of the packed path exists before the decode captures (the compile pass of the previous commit). Action-Id: ds41/optimize/233
The generator built the MoE padding configs of the chunk, the warm-up prompt, the listed trace lengths and the packed layouts' totals before the decode capture, but not of a layout's segment lengths: a packable prompt that runs alone (the join test's references, which list no trace length; a queue of one in serving) built its config lazily, after the decode capture, where the decode replays overwrite it. In the join test the k=2 references of the same prompt drifted from their k=4 run (PCC 0.97 at 40 layers, another argmax at 7), and the traced 40-layer run hung in the first eager chunk after the c1 replays. The compile pass now runs one eager pack of every listed layout, not the largest alone, so no layout's first run follows a capture. The test asserts the references stable across the decode replays and segment 0 of every pack equal to the kmax pack's. Action-Id: ds41/optimize/233
The join test read the candidate source's block pages whole, so the last partial page carried the slot's previous occupant's rows past the prompt's length: fresh slots agreed in the first phase, reused ones differed in the next (446 of 1,024 elements of page 128 of 129), as the single path's slots would. The pages are now decoded to rows and cut at the prompt's rows. The c1 token comparison is reported rather than asserted: the reference and the packed slot step alone at different pool steps, their ring rows at different physical slots, so the window's reduction order differs by rotation and a near-tie of the 7-layer model flips at step 3; the same-step comparison is the bit-exact gate. And V41_PJ_ROWS=0 skips the pool-row download: the test reads every pool tensor of every chip to the host, hundreds of GB at the served 1M context with 32 slots, where the OOM killer ended it. Action-Id: ds41/optimize/233
prefill_packed filled the pack's position tables with its own unbatched copies after _stage; the current _stage fills a generic state's tables itself (TT_V41_PREFILL_HOST_BATCH=1 puts them in the one dispatch round), and it takes the pack's layout since the rebase, so the packed path uses it too. Action-Id: ds41/optimize/233
The packed prefill state was built on the full-chunk model, which is 10,240 rows now: a pack of four 1,028-token prompts would pay that chunk (~0.65 s) where measurements showed 0.29 s on the 5,120 one, no better than the four single prefills it replaces (0.18 s each on the 2,048 geometry). TT_V41_PACK_CHUNK names one of TT_V41_SHORT_CHUNK's geometries to build the state on; the layouts are checked against that chunk, and the warm-up's eager pack and captures run on that view. Unset, the full-chunk model as before. Action-Id: ds41/optimize/233
A packed chunk (TT_V41_PACK) runs several whole prompts side by side, each at its own positions, but the chunk's DSpark seeding wrote the chunk's last window positions into the state's one ring set as if it were one prompt, so a packed segment could not join a decode slot with the drafter on (TT_V41_SPEC): the generator asserted on it. The packed state now holds one ring set per segment (V41PrefillState.segment_rings[g], allocated with the segment carries by pack_segments=), the chunk tables keep a seed-row RoPE pair per segment (window + 64 rows: a segment's offset is ratio-aligned, not tile-aligned) filled per layout with the pack's other host inputs (prepare_pack_host), and TtV41DSpark.seed_packed projects the taps gathered once at each segment's tile-aligned rows and writes that segment's last min(length, window) positions into its own rings at slots p % window: row-local throughout, so a segment's rings equal the single prefill's of the same prompt. seed_rows takes the chunk-row offset; seed() is unchanged in effect. prefill_packed slices the pairs inside the forward, so V41PackTrace captures the slice and a replay reads the host-filled buffers. The generator's join loads a segment's rings into the slot's drafter rings at the segment's length (_load_rings, shared with the single path), and the warm-up's pack joins do the same so the warm-up rounds draft from coherent rings. test_v41_pack_join.py compares each slot's drafter ring rows of the prompt's last 128 positions after the join (bit-equal for segment 0) and records the device rounds each slot took under the speculative round, which must agree between the single and the packed prefill of one prompt. Action-Id: ds41/optimize/234
TtV41VerifyDecoder.join overrides TtV41BatchDecoder.join without the segment argument the packed chunk's join takes (TT_V41_PACK), so the first segment join with the drafter on (the warm-up's) raised a TypeError. The override takes and forwards it; a packed segment is ratio-aligned and brings no compressor carry, so the multi-slot carry row below it is untouched. Action-Id: ds41/optimize/236
Two defects of the pack-join test under the served stack. The pool's window rings are 160 position-based slots with the verify decoder (TT_V41_SPEC), and a join of a 1,028-token prompt writes its last 128 positions at q % 160, leaving 32 rows to an earlier occupant; the test read the slot's first 128 ring rows blindly, so a replayed pack's slot and an eager pack's slot differed in the rows a previous decode wrote (68..80 of 128) while every written row was equal. The rows are now the prompt's last 128 positions at their slots (the whole ring where the slot is the pool step's). And the single reference's geometry is judged against the pack state's chunk (TT_V41_PACK_CHUNK's view), not the model's: on the served geometry the reference prefills eagerly on the 10,240 chunk while the pack sits on the 5,120 view, so segment 0 is reported there and asserted where the two share a chunk. Action-Id: ds41/optimize/238
tests/v41/test_v41_round_profile.py drives the generator as the plugin's ragged contract does (after test_v41_spec_generator.py): the two-phase warm-up, V41_RP_LIVE requests prefilled from distinct prompts of V41_RP_PROMPT tokens, then warm and measured rounds, one decode_forward a round. V41_RP_PROFILE=1 reads the device profiler after the warm-up, the prefills and every round, so each measured round is one replay session of its capture in the C++ report. V41_RP_OPMAP records, while the captures are made, which model function issued which device operation ids, so a replayed op is attributed by its id. Test only: nothing the server runs is changed. Action-Id: ds41/optimize/253
V41_RP_DRAIN_SKIP names the reads to skip (warm-up, prefills, ...). With the raw device log written, every read dumps what the buffers hold; a phase that is not read only overflows them and its markers are dropped, which bounds the log to the rounds that are measured. Test only. Action-Id: ds41/optimize/256
… one chain The verify step selected each live tile row's top-k among its candidate blocks with its own chain of 25 programs (select_user_in_blocks: the row's query repeated over 32 rows, candidate_scores, a top-k of a tile with one real row, the id arithmetic), 4 chains a consumer source in the one-slot round and 16 in the 32-slot one, every program at the launch floor whatever the row. TtV41Indexer.select_rows_in_blocks runs one chain for all of them: candidate_scores takes the live rows as (tile row, block base) pairs and ids of s rows per live row, reads each row's query, head weights and pin from its own tile row and its blocks from its own user's pages (QSPLIT in the reader and the writer; the per-row launches are unchanged), so one launch scores every live row's blocks on up to all cores; the scores are a row per live row again, one top-k and one id chain work on those rows. TtV41VerifyDecoder._index takes it under V41_VR_MERGE_BLOCKS=rows (default row: the per-row chain) and hands it the candidate source's stacked blocks without slicing them apart. No constant is allocated. test_v41_select_rows_in_blocks: the merged selection against the per-row one, bit-equal on every chip, in the one-slot and the 32-slot layouts. Action-Id: ds41/optimize/263
The round's record carried every row's emitted count; with the tokens two sessions on the same prompts are compared token for token (a flag's exactness on 32 distinct prompts). Test only. Action-Id: ds41/optimize/263
The verify step scored each densely scored live tile row with its own range_scores launch (32 cores, read-bound, most of a launch at a short context the fixed -inf fill of its row of the pool's width) and then concatenated the rows' split layouts for the merged selection: 4 launches a source in the one-slot round, 16 in the 32-slot one, and a copy of every score. range_scores(live=[(user, row), ..]) scores them in one launch that writes the stacked layouts itself: a unit is one split row of one live row (key user, split row, query row, output row), the units go round the first min(L * rows, grid) cores, and the three kernels work a core's units one after another (the reader keeps every unit's bound before it hands the pin page's buffer over; the writer's two chunk buffers alternate across units). The single launch is one unit a core, as before. TtV41VerifyDecoder._index takes it under V41_VR_MERGE_SCORES=rows where the merged selection runs (default row: a launch a row and the concat). test_v41_range_scores_rows: the one launch against a launch a row, bit-equal on every chip, 4 and 16 live rows, one and 32 split rows. The consumers' test shares its inputs through helpers. Action-Id: ds41/optimize/265
For a sentinel block id the reader read the slot's block 0 (finite values the writer masks to -inf). On a chip whose rows are all idle, the 28 padding chips of a one-slot step, every id is the sentinel and every core reads that one page K times: 86.6 us a launch there against 20 on the live rank's chips, whose ids are spread over the pool. V41_CANDSCORE_SENTINEL=skip issues no read for a sentinel id: the block's scores are whatever the buffer held and the writer masks them as before (a block's scores depend on its own rows only). Default read. test_v41_candidate_scores_skip_sentinel: the selections bit-equal between the two, rows with sentinel tails and idle rows; a launch over idle rows timed both ways. Action-Id: ds41/optimize/266
The 32-slot verify gate hung the device with range_scores(live=) on: 16 live rows a rank on a ratio-4 source of the 1,054,720-row pool, 263,712 columns in 32 split rows of 33 chunk slots, the 300-token prompt in every slot. test_v41_range_scores_rows checks the same 16 rows on a pool of 20,480 rows, three chunk slots a split row. test_v41_range_scores_rows_wide runs the one launch against a launch a row at the production width, 4 and 16 live rows, with the gate's lockstep positions (one non-empty split row a live row) and with the spread ones; the keys past row 20,480 are zero. Action-Id: ds41/optimize/268
With 32 split rows, 4 live rows (128 units) and 16 (512 units) both launch on the grid's first 120 cores with the same compile-time arguments, so they share a program-cache entry. On a hit the generic op re-applies each core's runtime arguments only up to the length the first call gave that core. The 32-slot round's verify step after its plain step makes exactly that sequence, and hung the device. test_v41_range_scores_rows_cache calls the one launch with the 4 rows of the one-slot layout, then the 16 of the 32-slot layout, then the 4 again, each against a launch a row. Action-Id: ds41/optimize/270
…ngth The generic op's program-cache key holds each kernel's cores and compile-time arguments but not each core's runtime-argument count, and a cache hit re-applies a core's arguments only up to the length the entry's first call gave it. range_scores(live=) gave a core 3 words a unit: 4 live rows (1 or 2 units a core on 120 cores) then 16 (4 or 5) reused the first call's program with the later units cut off, the reader, compute and writer disagreed on each unit's chunks, and the 32-slot verify step hung the device. Every core's reader, writer and compute arguments are now padded to RANGE_MAX_UNITS units, so the length is one for all calls and a hit carries every unit. The kernels are unchanged: they read n_units units. Action-Id: ds41/optimize/270
The three flags pass every exactness gate of the verify round (one slot at K = 1, 2, 3, 32 slots, the ring's wrap, the 1M pool's end) and pay their estimates at 32K: of verify device a round, the one-slot round 68.7 -> 65.0 ms and the 32-slot round 105.1 -> 89.7, the emitted tokens equal. V41_VR_MERGE_BLOCKS and V41_VR_MERGE_SCORES now default to rows and V41_CANDSCORE_SENTINEL to skip, so the served configuration takes them without a catalog line; row, row and read keep the per-row paths. Action-Id: ds41/optimize/274
| for u in rng.sample(range(4), blocks): | ||
| p = rng.randint(0, pmax) | ||
| a = rng.randint(0, 4) | ||
| taps = [(rng.randint(0, db.ROWS - 1), p - a + j) for j in range(a + 1)] |
There was a problem hiding this comment.
❗Cycode: SAST violation: 'Usage of weak pseudo-random number generator (Python)'.
Risk Score: 64 (MEDIUM)
Severity: High
Description
The random module in Python uses the Mersenne Twister algorithm, a pseudorandom number generator (PRNG) whose output is deterministic and reproducible. Given enough observed values, an attacker can reconstruct the internal state and predict future and past outputs. Because code evolves and values generated by random may flow into security-sensitive contexts over time (e.g., identifiers, tokens, session keys, or access-control decisions), using a predictable source of randomness anywhere in the codebase increases the attack surface.
Cycode Remediation Guideline
✅ Do
- Do use the
secretsmodule for generating random values. It provides cryptographically secure randomness, making it a safe default regardless of how the values are used downstream. - Do use
os.urandom()orrandom.SystemRandom()when you need a secure source of randomness compatible with therandomAPI.
❌ Don't
- Do not use
randomfor generating values that may be exposed externally or used in security-relevant logic, including identifiers, tokens, nonces, passwords, or access-control decisions. - Do not treat the usage context of
randomvalues as fixed — over time, refactoring and feature growth can expose previously internal values to security-sensitive flows.
📋 References
| a = rng.randint(0, 4) | ||
| taps = [(rng.randint(0, db.ROWS - 1), p - a + j) for j in range(a + 1)] | ||
| taps = [(t, q) for t, q in taps if 0 <= q <= p] if rng.random() < 0.9 else [] | ||
| es.append({"user": u, "token": rng.randint(0, 100000), "position": p, "taps": taps}) |
There was a problem hiding this comment.
❗Cycode: SAST violation: 'Usage of weak pseudo-random number generator (Python)'.
Risk Score: 64 (MEDIUM)
Severity: High
Description
The random module in Python uses the Mersenne Twister algorithm, a pseudorandom number generator (PRNG) whose output is deterministic and reproducible. Given enough observed values, an attacker can reconstruct the internal state and predict future and past outputs. Because code evolves and values generated by random may flow into security-sensitive contexts over time (e.g., identifiers, tokens, session keys, or access-control decisions), using a predictable source of randomness anywhere in the codebase increases the attack surface.
Cycode Remediation Guideline
✅ Do
- Do use the
secretsmodule for generating random values. It provides cryptographically secure randomness, making it a safe default regardless of how the values are used downstream. - Do use
os.urandom()orrandom.SystemRandom()when you need a secure source of randomness compatible with therandomAPI.
❌ Don't
- Do not use
randomfor generating values that may be exposed externally or used in security-relevant logic, including identifiers, tokens, nonces, passwords, or access-control decisions. - Do not treat the usage context of
randomvalues as fixed — over time, refactoring and feature growth can expose previously internal values to security-sensitive flows.
📋 References
| p = rng.randint(0, pmax) | ||
| a = rng.randint(0, 4) | ||
| taps = [(rng.randint(0, db.ROWS - 1), p - a + j) for j in range(a + 1)] | ||
| taps = [(t, q) for t, q in taps if 0 <= q <= p] if rng.random() < 0.9 else [] |
There was a problem hiding this comment.
❗Cycode: SAST violation: 'Usage of weak pseudo-random number generator (Python)'.
Risk Score: 64 (MEDIUM)
Severity: High
Description
The random module in Python uses the Mersenne Twister algorithm, a pseudorandom number generator (PRNG) whose output is deterministic and reproducible. Given enough observed values, an attacker can reconstruct the internal state and predict future and past outputs. Because code evolves and values generated by random may flow into security-sensitive contexts over time (e.g., identifiers, tokens, session keys, or access-control decisions), using a predictable source of randomness anywhere in the codebase increases the attack surface.
Cycode Remediation Guideline
✅ Do
- Do use the
secretsmodule for generating random values. It provides cryptographically secure randomness, making it a safe default regardless of how the values are used downstream. - Do use
os.urandom()orrandom.SystemRandom()when you need a secure source of randomness compatible with therandomAPI.
❌ Don't
- Do not use
randomfor generating values that may be exposed externally or used in security-relevant logic, including identifiers, tokens, nonces, passwords, or access-control decisions. - Do not treat the usage context of
randomvalues as fixed — over time, refactoring and feature growth can expose previously internal values to security-sensitive flows.
📋 References
| es = [] | ||
| for u in rng.sample(range(4), blocks): | ||
| p = rng.randint(0, pmax) | ||
| a = rng.randint(0, 4) |
There was a problem hiding this comment.
❗Cycode: SAST violation: 'Usage of weak pseudo-random number generator (Python)'.
Risk Score: 64 (MEDIUM)
Severity: High
Description
The random module in Python uses the Mersenne Twister algorithm, a pseudorandom number generator (PRNG) whose output is deterministic and reproducible. Given enough observed values, an attacker can reconstruct the internal state and predict future and past outputs. Because code evolves and values generated by random may flow into security-sensitive contexts over time (e.g., identifiers, tokens, session keys, or access-control decisions), using a predictable source of randomness anywhere in the codebase increases the attack surface.
Cycode Remediation Guideline
✅ Do
- Do use the
secretsmodule for generating random values. It provides cryptographically secure randomness, making it a safe default regardless of how the values are used downstream. - Do use
os.urandom()orrandom.SystemRandom()when you need a secure source of randomness compatible with therandomAPI.
❌ Don't
- Do not use
randomfor generating values that may be exposed externally or used in security-relevant logic, including identifiers, tokens, nonces, passwords, or access-control decisions. - Do not treat the usage context of
randomvalues as fixed — over time, refactoring and feature growth can expose previously internal values to security-sensitive flows.
📋 References
| entries = [] | ||
| for _r in range(8): | ||
| es = [] | ||
| for u in rng.sample(range(4), blocks): |
There was a problem hiding this comment.
❗Cycode: SAST violation: 'Usage of weak pseudo-random number generator (Python)'.
Risk Score: 64 (MEDIUM)
Severity: High
Description
The random module in Python uses the Mersenne Twister algorithm, a pseudorandom number generator (PRNG) whose output is deterministic and reproducible. Given enough observed values, an attacker can reconstruct the internal state and predict future and past outputs. Because code evolves and values generated by random may flow into security-sensitive contexts over time (e.g., identifiers, tokens, session keys, or access-control decisions), using a predictable source of randomness anywhere in the codebase increases the attack surface.
Cycode Remediation Guideline
✅ Do
- Do use the
secretsmodule for generating random values. It provides cryptographically secure randomness, making it a safe default regardless of how the values are used downstream. - Do use
os.urandom()orrandom.SystemRandom()when you need a secure source of randomness compatible with therandomAPI.
❌ Don't
- Do not use
randomfor generating values that may be exposed externally or used in security-relevant logic, including identifiers, tokens, nonces, passwords, or access-control decisions. - Do not treat the usage context of
randomvalues as fixed — over time, refactoring and feature growth can expose previously internal values to security-sensitive flows.
📋 References
| def _instrument(block, sink, mesh_device): | ||
| def wrap(owner, attr, name): | ||
| if getattr(owner, attr, None) is not None: | ||
| setattr(owner, attr, _Timed(getattr(owner, attr), name, sink, mesh_device)) |
There was a problem hiding this comment.
❗Cycode: SAST violation: 'Unsanitized external input in code generation (Python)'.
Risk Score: 74 (HIGH)
Severity: Critical
Description
Allowing external input (dynamic or user-controlled) to directly influence code generation or scripting functions without proper sanitization can lead to code injection vulnerabilities. This occurs when an attacker is able to insert malicious code into your application, which is then executed, potentially leading to unauthorized actions or data access.
Cycode Remediation Guideline
✅ Do
- Do always validate or sanitize input to ensure it does not contain harmful code before using it in such contexts.
❌ Don't
- Do not pass unsanitized external input to functions or methods that dynamically execute code.
📋 References
| for _r in range(8): | ||
| es = [] | ||
| for u in rng.sample(range(4), blocks): | ||
| p = rng.randint(0, pmax) |
There was a problem hiding this comment.
❗Cycode: SAST violation: 'Usage of weak pseudo-random number generator (Python)'.
Risk Score: 64 (MEDIUM)
Severity: High
Description
The random module in Python uses the Mersenne Twister algorithm, a pseudorandom number generator (PRNG) whose output is deterministic and reproducible. Given enough observed values, an attacker can reconstruct the internal state and predict future and past outputs. Because code evolves and values generated by random may flow into security-sensitive contexts over time (e.g., identifiers, tokens, session keys, or access-control decisions), using a predictable source of randomness anywhere in the codebase increases the attack surface.
Cycode Remediation Guideline
✅ Do
- Do use the
secretsmodule for generating random values. It provides cryptographically secure randomness, making it a safe default regardless of how the values are used downstream. - Do use
os.urandom()orrandom.SystemRandom()when you need a secure source of randomness compatible with therandomAPI.
❌ Don't
- Do not use
randomfor generating values that may be exposed externally or used in security-relevant logic, including identifiers, tokens, nonces, passwords, or access-control decisions. - Do not treat the usage context of
randomvalues as fixed — over time, refactoring and feature growth can expose previously internal values to security-sensitive flows.
PR Category
Feature: bring-up of Deepseek v4.1 on a single Galaxy.
Summary
This is a Kernelize project to upstream Tenstorrent support for Deepseek v4.1. It builds on work by @mvasilijevicTT but reworks significant portions to substantially improve prefill TTFT. It has accompanying PRs in other repositories: tenstorrent/vllm-tt-plugin#340, tenstorrent/tt-inference-server#5586.
Notes for reviewers
@bgoelTT, as requested, this branches off of v0.76.0; this means it is considerably out of date with
main. Also, it adds 156 commits above those used from @mvasilijevicTT and not all of those need to go in this PR. There are all kinds of bug fixes, debugging helpers, tests, etc., that could become separate PRs for easier review. Let me know how you want to proceed.Verification
Code owners